Selecting More Informative Training Sets with Fewer Observations

Abstract A standard text-as-data workflow in the social sciences involves identifying a set of documents to be labeled, selecting random sample them label using research assistants, training supervised learner remaining documents, and validating that model’s performance accuracy metrics. The most resource-intensive component this is hand-labeling: carefully reading paying human coders duplicate or more. We show hand-coding an algorithmically selected rather than simple-random can improve model above baseline by as much 50%, reduce costs up two-thirds, applications predicting (1) U.S. executive-order significance (2) financial sentiment on media. accompany manuscript with open-source software implement these tools, which we hope make learning cheaper more accessible researchers.